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I s the time allowed for National Assessment Program Literacy and Numeracy 
(NAPLAN) numeracy tests sufficient? Is there any evidence that students 
run out of time and, if so, what are the implications for teachers who prepare 
students for NAPLAN numeracy tests? 

In each of the 2010 Years 7 and 9 numeracy tests students were required 
to complete 32 test items in 40 minutesa n average of 75 seconds per ques¬ 
tion. Norton (2009, p. 28) noted that time limitations were an issue in the 2008 
Year 9 NAPLAN numeracy tests, but did not seek to examine their effect. This 
paper analyses the available quantitative student test response data to exam¬ 
ine whether there is evidence of a timing issue for students. 


Quantitative data 


The only way to accurately determine how many students did not have enough 
time to complete the NAPLAN numeracy tests is to ask the students soon after 
completing the tests. In the absence of this information, it was necessary to 
make judgements based on the students responses to test items. 

The study focussed on those items where students did not record a response. 
There were many reasons why students might not have responded to a test 
item, for example, an oversight, the complexity of the item, the nature of the 
response required (multiple-choice or student-determined response), and the 
motivation of the student. To avoid capturing some of these cases, the study 
assumed that a student had problems with a lack of time if he/she failed to 
respond repeatedly to items in the last part of the test. Accordingly, out of 
time” was defined to be a failure to respond to at least three of the last ten 
items in the test. 

It is recognised that this assumption is critical to the findings of the study. 
As most teachers advise students to guess if they cannot respond in any other 
way, it is considered that the most likely reason for failing to respond to several 
items is lack of time. It is acknowledged that some students fail to respond to 
many of the items that are not multi-choice. Such students would be captured 
in the definition of “out of time”. On the other hand, students not included 
were those who, aware that time was running out, rushed or guessed several 
responses in order to complete all items. 

The study accessed data that showed students responses to each test item 
in both numeracy tests. This information was made available to schools (for 
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their own students) in the form of a csv electronic file, but was not given to 
any other groups. Accordingly, this study was based on information supplied 
by a small number of Queensland schools who agreed to be part of the study. 
Of necessity, this was not a random sample. Logistic regression techniques 
were used to allow for the potentially biased nature of the sample, allowing the 
result to be extended beyond the schools in the study. A dichotomous variable 
out of time was assigned values of 1 if the condition of failing to answer three 
of the last ten questions was satisfied and 0 otherwise. 


Year9 

The Year 9 data used in the study were available for a total of 947 students in 
four Queensland schools, both Government and non-Government. Of these, 
70 (7.4%) ran out of time in the non-calculator test, 117 (12.4%) ran out of 
time in the calculator test, with 140 (14.8%) running out of time in at least 
one test. 

Naturally, these results would be influenced inversely by the ability of the 
students. Given the accessibility nature of the sample, the students included 
in the study may not have been typical of others. To overcome the effect of any 
bias in the sample, the study sought to develop a logistic regression model 
that calculated the probability of being out of time for any given NAPLAN 
numeracy scale score 1 , labelled NumScore in the analysis. The results are 
summarised in Table 1. 

By substituting the 
mean NAPLAN numeracy 
scale score for all Year 9 
Queensland students into 
the model, the probability of 
a typical student running out of time was estimated. The analysis suggest¬ 
ed that there was a probability of approximately 0.17 that a student with a 
NAPLAN numeracy scale score of 578 (the Queensland mean) would run out of 
time in one or both NAPLAN numeracy tests. 

It was hypothesised that running out of time may also be influenced by 
reading ability. Accordingly, similar analysis was conducted using both the 
NAPLAN reading and numeracy scale scores as independent variables in the 
logistic regression model. However, the reading scale score did not prove to be 
significant. 


Table 1. Results of logistic regression for"out of time" data, Year9. 



B 

&E. 

Wald 

df 

Sig. 

Exp(B) 

NumScore 

-0.007 

0.001 

29.439 

1 

0.000 

0.993 

Constant 

2.438 

0.760 

10.296 

1 

0.001 

11.449 


Year7 

The study also had access to similar data for Year 7 students in two schools. 
Unfortunately, the regression analysis of the Year 7 results similar to that 
undertaken for the Year 9 results yielded unreliable outcomes. However, one 
of the schools, with 209 students in Year 7, had a mean NAPLAN numeracy 
scale score of 545, the same as the Queensland mean. In this school, 22% of 
students ran out of time. Naturally, caution must be exercised in relying on 
the results of a single school, albeit with a large Year 7 enrolment. 


1 Details of the meaning of scale scores can be found at the MySchool website (Australian 
Curriculum Assessment and Reporting Authority, 2010b). 


amt 68(1) 2012 


37 














Discussion 


Practicality requires that tests are of a finite duration and there will always be 
some students who do not complete the tests in the available time. Whilst it 
becomes a matter for judgement as to whether a given non-completion rate is 
unacceptable, it is suggested, that a figure of up to 17%, as suggested by this 
study, is too high. The content of the 2010 NAPLAN numeracy tests was based 
on the Statement of Learning for Mathematics (Curriculum Corporation, 2006). 
There is no requirement in this statement that students should work under 
time pressure. It is submitted, therefore, that it is unreasonable to impose this 
condition on one out of every six students in NAPLAN numeracy tests. In these 
circumstances, a non-completion rate of less than 10% might be acceptable. 


Selected NAPLAN Questions 

It not possible in an article of this length to analyse every question in the 
NAPLAN numeracy tests in detail. However, one example may serve to illus¬ 
trate the nature of the problem. 

Question 9 in the 2010 Year 7 calculator test (Australian Curriculum 
Assessment and Reporting Authority, 2010a) provided students with a two- 
dimensional table showing the number of people attending sport training on 
each weekday of a four week period. Students were asked to find which day 
of the week had the greatest total attendance. The options offered to students 
were Monday, Tuesday, Wednesday and Thursday. 

The author observed 10 Year 7 students of varying abilities complete this 
question. Nine of the students chose to add all four rows using their calculator, 
and then select the largest value. Only one student attempted to use estima¬ 
tion to find the answer (adopting an incorrect method). One other student 
recognised that the total for Thursday had to be less than the total for at least 
one of other days, but chose to add the row anyway. When another student 
was asked why he did not use estimation, he said, I like to be certain. 

Questions that test simple concepts should be quick for students to answer 
so that they can accumulate the additional time needed for more complex 
questions. As a result of the time-consuming method used by nine of the 
students, this relatively straightforward question took them significantly more 
than the average of seventy five seconds available for each question. Whilst 
74% of all students in Australia answered this question successfully, it must 
be asked whether the amount of time taken to respond to this question affect¬ 
ed their success in later questions. 

If the purpose of the question was to test the ability of students to identify the 
need for, and use, appropriate estimation techniques, then the question failed 
to elicit the desired response in any of the 10 students observed. There are two 
possible reasons for the fact that nine of the 10 students did not attempt to use 
estimation techniques. Firstly, Year 7 students may not have thought to look for 
alternative approaches to a question in order to select the most time-efficient 
method. Secondly, estimation may not have been seen as a legitimate technique 
in a test where the availability of calculators implied that accuracy was important. 

Questions like this one, that many students would attempt to answer by 
testing all four of the available options, typically take longer than questions 
where students can calculate the answer directly. Table 2 lists similar ques¬ 
tions in the 2010 tests. 

Test writers should consider whether the same concepts could have been 
assessed in less time-consuming ways. 
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Table 2. Possible time-consuming questions in the 2010NAPLAN numeracy tests. 


Test 

Question 

Details 

7 non-calc 

25 

Testing different arrangements of 4 digits in a 3 digit by 1 digit product to give a 
specified result 

7 calc 

14 

Testing which of four divisors (43, 45, 47, 49) of 4465 givesa whole numberquotient 


25 

Matching four pairs of results in a table to fourdifferent linearequations 

9 non-calc 

5 

Testing 4 sets of 3 numbers to satisfy an inequality with 3 independent variables 


21 

Examining fourdifferentalgebraic expressionsto determine which results in the 

largest value when a specified variable is less than one 

9 calc 

17 

Finding the shortest path in a network of 6 possible paths 


18 

Same asquestion 25 in the Year7 calculatortest 


Implic a ions for teac hers 

The main purpose of this article is to alert teachers to the possibility that lack 
of time may be a concern for students in NAPLAN numeracy tests. Given the 
limitations of the data available to this study, teachers should question their 
own students and examine the NAPLAN numeracy results to determine if time 
limitations cause problems for them. Class groups of students could be ques¬ 
tioned about the adequacy of the time available after practice tests (a show of 
hands would be sufficient) 2 . Data hies showing the results from past NAPLAN 
numeracy tests and the responses of students to NAPLAN practice tests could 
be examined for evidence of students running out of time. Whilst most schools 
would not have the resources to undertake logistic regression, they should 
be able to make judgements that take into account the abilities of their own 
students. 

There are several strategies that teachers could use to assist students to 
work more quickly. 

• Improving mental arithmetic skills. A calculator is a time consuming alter¬ 
native to mental computations. Norton (2009, p. 27), analysing the 2008 
Year 9 NAPLAN calculator allowed test, found that the use of a calculator 
would have been of significant assistance for only two of the questions, 
assuming that students possessed reasonable computation ability . 

• When teaching problem solving, teachers should encourage students to 
consider whether there is more than one method of responding to a ques¬ 
tion. Estimation should be presented as a legitimate strategy when elimi¬ 
nating multiple choice options or in other situations when accuracy is not 
critical. It can be rehearsed regularly in the context of problem solving as 
a method of verifying solutions. Marking schemes for school assessment 
should accept estimation where appropriate. 

• When preparing students for NAPLAN numeracy tests, teachers must 
encourage students to take account of the amount of time available to them. 
Where possible, students should recognise that several strategies could be 
used to solve a particular problem and identify those that are the most 
time-efficient. The aim is to encourage students to consider more than one 
strategy for answering a NAPLAN numeracy test question and in order to 
select the most rapid method. For Year 7 students in particular this may 
require considerable practice. 


2 One of the schools involved in this study undertook this show of hands analysis immediately 
after the 2011 NAPLAN numeracy tests and found that 30.5% of 204 Year 9 students and 
15.2% of 158 Year 7 students indicated that they needed more time in at least one of the 2011 
numeracy tests. 
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Implications for NAPLAN test developers 


The adequacy, or otherwise, of the time available to students in NAPLAN 
numeracy has important implications for the validity of NAPLAN numeracy 
data. It would be of relevance to all teachers of numeracy to know what the 
Australian Curriculum Assessment and Reporting Authority (ACARA) consid¬ 
ers an acceptable level of non-completion of the tests, and this information 
should be made available to teachers. 

The unavailability of detailed NAPLAN data to anyone other than the school 
concerned makes it impossible for anyone outside the relevant government 
agencies to rigorously analyse of the effect of time limitations on students 
attempting NAPLAN numeracy tests. Accordingly, it is recommended that 
ACARA undertake a quantitative study, using the results of all students or of 
a randomly selected sample of students, to determine how time limitations in 
the various NAPLAN numeracy tests affect students ability to show what they 
can do. As the entire community has a legitimate interest in this information, 
the results of the research should be published on the ACARA website. If the 
research indicates that students do not have sufficient time to show what they 
know and can do in NAPLAN numeracy tests, ACARA must take account of 
these findings when developing future NAPLAN numeracy tests. 

Whilst this article has focussed on the NAPLAN numeracy tests, there 
would be value in conducting a similar analysis of NAPLAN testing in the other 
domains. 


Conclusion 

There are indications that some students may not have had sufficient time to 
show what they could do in the 2010 Year 7 and 9 NAPLAN numeracy tests. 
Despite limitations of the lack of data available to this study, there is enough 
evidence to alert all schools that lack of time in Year 7 and 9 NAPLAN numer¬ 
acy tests could be a concern for some students. Schools should investigate 
whether this applies to their own students. If time limitations have an adverse 
impact on significant numbers of students, then this has important implica¬ 
tions for teachers preparing students for NAPLAN tests and for the developers 
of the tests. 
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